{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPdQvYmlWmNc"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/llama-2/llama-2-13b-retrievalqa.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/llm-field-guide/llama-2/llama-2-13b-retrievalqa.ipynb)\n",
        "\n",
        "# RAG with LLaMa 13B\n",
        "\n",
        "In this notebook we'll explore how we can use the open source **Llama-13b-chat** model in both Hugging Face transformers and LangChain.\n",
        "At the time of writing, you must first request access to Llama 2 models via [this form](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) (access is typically granted within a few hours). If you need guidance on getting access please refer to the beginning of this [article](https://www.pinecone.io/learn/llama-2/) or [video](https://youtu.be/6iHVJyX2e50?t=175).\n",
        "\n",
        "---\n",
        "\n",
        "\ud83d\udea8 _Note that running this on CPU is sloooow. If running on Google Colab you can avoid this by going to **Runtime > Change runtime type > Hardware accelerator > GPU > GPU type > T4**. This should be included within the free tier of Colab._\n",
        "\n",
        "---\n",
        "\n",
        "We start by doing a `pip install` of all required libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K_fRq0BSGMBk",
        "outputId": "c5e0880c-08e7-4ac3-dc7f-41a7c41df954"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.4/7.4 MB\u001b[0m \u001b[31m14.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m86.0/86.0 kB\u001b[0m \u001b[31m8.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m179.1/179.1 kB\u001b[0m \u001b[31m15.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m492.2/492.2 kB\u001b[0m \u001b[31m19.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m244.2/244.2 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m109.1/109.1 MB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m92.6/92.6 MB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m268.8/268.8 kB\u001b[0m \u001b[31m28.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m97.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m71.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m78.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m60.0/60.0 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m300.0/300.0 kB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m212.5/212.5 kB\u001b[0m \u001b[31m25.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m90.0/90.0 kB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m3.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m74.5/74.5 kB\u001b[0m \u001b[31m4.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ],
      "source": [
        "!pip install -qU \\\n",
        "  transformers==4.31.0 \\\n",
        "  sentence-transformers==2.2.2 \\\n",
        "  pinecone-client==3.1.0 \\\n",
        "  datasets==2.14.0 \\\n",
        "  accelerate==0.21.0 \\\n",
        "  einops==0.6.1 \\\n",
        "  langchain==0.0.240 \\\n",
        "  xformers==0.0.20 \\\n",
        "  bitsandbytes==0.41.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fK7OXFdulxo6"
      },
      "source": [
        "## Initializing the Hugging Face Embedding Pipeline\n",
        "\n",
        "We begin by initializing the embedding pipeline that will handle the transformation of our docs into vector embeddings. We will use the `sentence-transformers/all-MiniLM-L6-v2` model for embedding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 465,
          "referenced_widgets": [
            "6c8b2df56ce64a5c9c48a17931ec82d2",
            "b0acbf492c0a434fb389f9d5458ab63b",
            "de4a0fb34611496cb4e4c0a903bfda3d",
            "ea00e7cfc0b84f128bc40ae376fcd966",
            "f79a83b6fe1d465988e895b26d770bae",
            "2b90feffb6bd4d658ec2e4e527298b85",
            "dee59810e1f04442a5d2f6da605d7fdc",
            "a222a53feb214b54bf3b5d15bfc1067c",
            "5afd3d03cd3f4e38be003ae7979bf7ff",
            "647293d230c44b9f92a58c1f637238b2",
            "d0c21f4c75064d1b87261a83b6f418cd",
            "8884e134c8af415aa5608bc2ba0aef02",
            "030e0975914740faa10f2b5339077174",
            "d57c250f20b543d79a4441c0bb7925c2",
            "de53022d5b924f8391be1cf03cdd0f34",
            "988612a625454d608b645a621c15e23b",
            "40671c3c511c4e04bf03104f6760afd6",
            "e6271c66dc6542d6b201d2e7f805cad1",
            "567fc77227a44659ad2221b749680977",
            "a46d368933f64a02ab7ba1f1a6db3930",
            "1130e7071bf74564b374294b41074e42",
            "ec15770732b241bd864499eae8925d8b",
            "8849074a46dc4f0f8dd0e04a49165f92",
            "92f4d9e7b335418195d8a76888a987b1",
            "e300e209cc2b473180c0a65c06971e2d",
            "fcd4a3fd66ff42bb83560e1050a14d5b",
            "b47c3404d3fc445292622fe8f8ab5e03",
            "68c62a13e56449d99c8bb7ffd79b8933",
            "38bab25aafd6425a8da60d4acafdc522",
            "76c0f87a4ff04c1a9928cd2daeee7bf9",
            "ec80abfb26cf400c969c717ef56ebacc",
            "2b2520045a8c4128a68f37636d98ffad",
            "72e69e3988e048e4bc9d3d859305a056",
            "d1c45f6ec99241f98d3435ae2e254c90",
            "d3cc59a526624549a7bbcb28cc672145",
            "67b9bdd1a5d44adca1d25f4bedd24ddb",
            "ec475cbc2fd74f779cc45b299d9a9706",
            "b25aedbeffda460c9669009aecf24568",
            "4da892be30734bf2853abb38b9329655",
            "d7e85b46dd3b4a75ab44cd7a7a5eab91",
            "fb52766cb9de4327bd15fa7414d051e6",
            "5562d0049b99401e99921aff0def58a1",
            "d441a1ac44d1429d99c86320476fb587",
            "80958275163f4a98ba786171e077e1ba",
            "aa7fc091798f4214b6822b6c8271e43c",
            "b5d97617579f4517b09a2188cdc6714b",
            "694e4ab1c8ca43ebaeffc190e7875aea",
            "c0e82c4c057b4edc8deb45013e72ceed",
            "c1dd99da696148e2b04e812e4816c323",
            "3cefd834dff3468d928b5b2957ac0e54",
            "68eeef80f5654f7493f933c822bdc90d",
            "6428093a34724147a00fbc0e26b50c58",
            "05de3c5acb5d4b19b9e3be4adea3614a",
            "9a9089bbcf7f429993d2c57ea8e7ea3d",
            "817a028e4f834ca6ac81b7b006c6ec1f",
            "479ee9d29bfd4a8db83cbefd33cb7bc6",
            "ea2936792fe943b3b21bf83764f84057",
            "0eca4f1d8d9c4250956e6e9037fd08dc",
            "836a24caecff48849ac250d99fca2337",
            "f1c3a8a77add4f57beeb8e48f0586ec9",
            "3757adc8258747f88386d28a943b3a8a",
            "5fc750c14727471b96b6215875f4b5b4",
            "98b88e072d23479ba437b36efcd14c22",
            "8e1c16a90f8344daadf54e8dc1f5c33c",
            "b86c03dd7bd24d61aadd526229f9cc7d",
            "1b6502c1504d4956ae835ec89c7dc0bd",
            "f6441e43cfba48a18dd0f966c99cea9f",
            "2be673e4d77545ae96c007e7732896ee",
            "87809b270722450d954e84a9b46fbe76",
            "188d40a3f6b94bc6ab1af895020d4ae8",
            "86b47f319dfb45be8298d1474a230eca",
            "85ca8e493d5c45eda8e1de8f070158ba",
            "c95232780e504a5cb4bf17db51aee251",
            "1c9f9c9028814615a6815bf6750cc3f2",
            "93fadd1dba2f4e62849132ca66fff36c",
            "a96110c79ab24bdea1346f8e0e3608e6",
            "5e8ff4cb8bd847e4a7f7d0ea9cdc06e9",
            "1cbd8e1b277e490d8fbf8bb7a54f1b2b",
            "934897f499bc4fcc940e02a444cff751",
            "5997a3c87d8c490783fb07171c80ff05",
            "30bb77012d63479a8401821de76cf8b8",
            "6037310ca63d4723bfecc1041b031f23",
            "fdead558e72f44d5aa41f2e0dc6d8a42",
            "f9a20fce3ec94994840521f8cb1addcb",
            "749722f9ab694071a3a5e92b1e1cbaa7",
            "24dab28f43e44b19883d60278ebbb507",
            "b5cbfaa505674df29bcdcf4139ac7632",
            "fea1ba840bb441fd81ab83f2a3347df9",
            "9dbe451b2a294b738b942811ab941993",
            "ee9b5450358c4be3b50a8e69c00b8ac9",
            "342dd440c8d04d64858707bb43995198",
            "53d1acba331c47f0919c8718717a0db6",
            "f49753853c4c4af2ab9d99dc677c5c68",
            "344bc5c011cc4011b3fd03788b0c8ce2",
            "d634a5876ae144e18ee803e69cccd268",
            "a2123d95c74b4189ac940e928000b89c",
            "01e322228ef94a6fae9a52fa7caa9c66",
            "7f1feb97196647bd9d0c24b4e95d9da4",
            "79516084e51d4d01aea52704e31cd9de",
            "d4a14dac635f4b5e894653ca2f67135d",
            "2076466e0f15485ea8ddf2c2756fd3bb",
            "76f5d758916d463195073c2d4537cdec",
            "5ec68593493e49958f9b8906d43c015e",
            "bbb25be9142e41c49611003f46b07f9e",
            "339c9ae28927475f8b6c7136e272a3cb",
            "01912135469f4d609ad1e215c114444f",
            "359a9e76a9b049d6aae893f729d8986a",
            "683b5cdf1c30499a82c0fc6c717f0af0",
            "ad078acd99c847cb874e240a030c4a9a",
            "1e8be8ca70004d388f9bbc2ad66590a1",
            "38bca2a9c99a4cbabd53ab25a0eae46c",
            "f00e03f8af0b47b1ab9389728675e540",
            "a115d050ede2420ab932d9305ccbcf07",
            "ff66c2e8c9fd471b81a8d9769232b584",
            "378ef3198c4e48fd940b3dccde1afb08",
            "f85646df3acd485c9c6d20bd03eb1a04",
            "32e00619f8a94382860fb6771874ed15",
            "fde71baba9dd46088c02cf68bee53c85",
            "3155f97822eb47feacb1e41b738382b3",
            "32fd293935914707b3e2d2a1a721207f",
            "dd84988329974f7c8ffe1a69d5388df3",
            "1e25bcb331604bb6b13cb77cbe104899",
            "d51077b2bbbd4cce83c886aca99de985",
            "78bfeb71b9434831a90653e2fd3321bf",
            "f941610f0e2845b7a118078a8ce665eb",
            "32d75338666f4c2181abce726ef83d0b",
            "47649245e1954bd687b0eabbafc0fd09",
            "f90a2ce8a0f54f9d99b8b18e7baf66a7",
            "56cf63ae329c4a5fb269ad5bf2065a5d",
            "7722806627d1454ba64ceb07ef028c09",
            "d4d0b303c0a64836b5d90c671e128809",
            "695358daecbd476a8f1bd5fc93759761",
            "2c696ac9026b42628ee73a34029ab3f0",
            "9e6d18fe47704d8a9c6d8b6fb5a0c3a4",
            "b94ce7514eb04345912e24ef0f797dc5",
            "f4aaad1753f148e6909bf71f7b5889df",
            "0a8955c7aa994808bff99c6520f1ca95",
            "9c2f99777e5b4afaa4f134fe3d9a3453",
            "4cd335ea02824928a62369d2142a8f41",
            "afd155f007a34c139571af272643acc5",
            "9a6115d04e12417993215fb9b4311d9c",
            "fca6af727a0c4716bb89af6eef4facf3",
            "350ecbda41274dfe8fffe9e793dd4fbd",
            "32aefb889dcf4041af0c48d96fd5b56a",
            "c0ecee1bcb2444069b2f941fe95762e4",
            "48665df5c7fa4d47b09184d50ded70dc",
            "3b82f60530864b008a550faadf7f5fbc",
            "5d675fb301a645cf86e4b0e7490921da",
            "211c6aee28ab45b68937518d3792572a",
            "a650992ded3741369c8edadc5b37cec3",
            "b25454f4395d46f39cf7ba22c7eafa12",
            "2a05219192674c32b47e5eb091acffc7",
            "14ed556c467c4e24a9eb7583ba0b51ac",
            "9c6301f2874146e6afcb6b29b9eebe2a"
          ]
        },
        "id": "nQf0ICZXmGPq",
        "outputId": "1daa06e1-ad8e-4a6f-f347-56cf95653e59"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6c8b2df56ce64a5c9c48a17931ec82d2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)e9125/.gitattributes:   0%|          | 0.00/1.18k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8884e134c8af415aa5608bc2ba0aef02",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8849074a46dc4f0f8dd0e04a49165f92",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)7e55de9125/README.md:   0%|          | 0.00/10.6k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d1c45f6ec99241f98d3435ae2e254c90",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)55de9125/config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "aa7fc091798f4214b6822b6c8271e43c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)ce_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "479ee9d29bfd4a8db83cbefd33cb7bc6",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)125/data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f6441e43cfba48a18dd0f966c99cea9f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1cbd8e1b277e490d8fbf8bb7a54f1b2b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)nce_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9dbe451b2a294b738b942811ab941993",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d4a14dac635f4b5e894653ca2f67135d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)e9125/tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "38bca2a9c99a4cbabd53ab25a0eae46c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1e25bcb331604bb6b13cb77cbe104899",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)9125/train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2c696ac9026b42628ee73a34029ab3f0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)7e55de9125/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "32aefb889dcf4041af0c48d96fd5b56a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)5de9125/modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from torch import cuda\n",
        "from langchain.embeddings.huggingface import HuggingFaceEmbeddings\n",
        "\n",
        "embed_model_id = 'sentence-transformers/all-MiniLM-L6-v2'\n",
        "\n",
        "device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'\n",
        "\n",
        "embed_model = HuggingFaceEmbeddings(\n",
        "    model_name=embed_model_id,\n",
        "    model_kwargs={'device': device},\n",
        "    encode_kwargs={'device': device, 'batch_size': 32}\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YSNke_aDnho-"
      },
      "source": [
        "We can use the embedding model to create document embeddings like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_4uRQacunhDP",
        "outputId": "916be9d7-7b03-4d8a-9ef2-f11aa1e2f921"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "We have 2 doc embeddings, each with a dimensionality of 384.\n"
          ]
        }
      ],
      "source": [
        "docs = [\n",
        "    \"this is one document\",\n",
        "    \"and another document\"\n",
        "]\n",
        "\n",
        "embeddings = embed_model.embed_documents(docs)\n",
        "\n",
        "print(f\"We have {len(embeddings)} doc embeddings, each with \"\n",
        "      f\"a dimensionality of {len(embeddings[0])}.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E4SSLvJqqdhZ"
      },
      "source": [
        "## Building the Vector Index\n",
        "\n",
        "We now need to use the embedding pipeline to build our embeddings and store them in a Pinecone vector index. To begin we'll initialize our index, for this we'll need a [free Pinecone API key](https://app.pinecone.io/)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "lhXARZQXq6QD"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from pinecone import Pinecone\n",
        "\n",
        "# initialize connection to pinecone (get API key at app.pinecone.io)\n",
        "api_key = os.environ.get('PINECONE_API_KEY') or 'PINECONE_API_KEY'\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we setup our index specification, this allows us to define the cloud provider and region where we want to deploy our index. You can find a list of all [available providers and regions here](https://docs.pinecone.io/docs/projects)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from pinecone import ServerlessSpec\n",
        "\n",
        "cloud = os.environ.get('PINECONE_CLOUD') or 'aws'\n",
        "region = os.environ.get('PINECONE_REGION') or 'us-east-1'\n",
        "\n",
        "spec = ServerlessSpec(cloud=cloud, region=region)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MSoNo9uUrlK3"
      },
      "source": [
        "Now we initialize the index."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "index_name = 'llama-2-rag'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import time\n",
        "\n",
        "# check if index already exists (it shouldn't if this is first time)\n",
        "if index_name not in pc.list_indexes().names():\n",
        "    # if does not exist, create index\n",
        "    pc.create_index(\n",
        "        index_name,\n",
        "        dimension=len(embeddings[0]),\n",
        "        metric='cosine',\n",
        "        spec=spec\n",
        "    )\n",
        "    # wait for index to be initialized\n",
        "    while not pc.describe_index(index_name).status['ready']:\n",
        "        time.sleep(1)\n",
        "\n",
        "# connect to index\n",
        "index = pc.Index(index_name)\n",
        "# view index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyckdnprEQDT"
      },
      "source": [
        "With our index and embedding process ready we can move onto the indexing process itself. For that, we'll need a dataset. We will use a set of Arxiv papers related to (and including) the Llama 2 research paper."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 266,
          "referenced_widgets": [
            "ede90b58e6454921807ed5e4add14dfe",
            "10a6f297ed6d49419b41e81c92bbe0e3",
            "9423eff20bb549da95a7386ab70eabc0",
            "af3d48837f724738b670452de0abb113",
            "d316abada6034cff96a836859d3ca357",
            "71f8707463354e0c96b4501abdb33dd4",
            "31182245515b4dfb9f2d187e6d82b76c",
            "dbdd1c27e6bd45b080dd342e8a9e29e6",
            "90251ba2fadd48cb80bc3f6030027131",
            "ce8e13502dc24abcb9cf542350c714a4",
            "826617d53bb742b99833bea820a05ed6",
            "8bf58fdcd82d40399aaa8b89a65d3533",
            "aa967db8e77e456f964b0e9e809a4d93",
            "e744e71de10149008d2f73a59b7643b2",
            "b78223be1ccc460885f640ef27682d16",
            "028eb037988f4f1c85056245dfa36c98",
            "f7a8e52df3464acfbd6e41a5f6e4cbc8",
            "027da23d80e04484a5c185ebcb93c2f2",
            "8d2ba3a46c9e430cb9ba25bc3ba244d0",
            "1a8d502e0d184dcb8878f740b989c019",
            "469b81195ece436a9cd0451688b9204e",
            "f8be996c3f594602a883a292c3b073bf",
            "aeadb1c8a93d4c78a13eaa62b12b42ae",
            "0bff078c638747b08c461aa0ea657c68",
            "8870191daa074186bfc07352fbf907d5",
            "e6ef5f1de6f74960ba9b67f40ad1a90a",
            "39581f5e389a436ebdbf280d4bf1bc2f",
            "baeb6fc9622f41acba7cf22e87577f64",
            "d4ac474e45164f4f862cbde666d0b66a",
            "855bce5189ef469eb96ae53b9a677909",
            "923dec68ed924fcfb2c548380633e63b",
            "27decb63e37748f6b06d03be9796ba91",
            "7d6f0db875cd45109031ad135446fab9",
            "08a5d57f57bb415f96c3e297b35b669d",
            "0ea6adf8c08c48d987dab2514b6a9f98",
            "32a894e17ec8411ab538c7d6cec3904b",
            "c605f0d5325d421eadcd27a5e232deae",
            "3caaff6070f64ddc89bae90c7c916c92",
            "0f6ffe18f4f14d50abc3ad9b9acfaab4",
            "62d35f8920d94db388a0abb681ec1aee",
            "4454f899de444612bf79b6b6531cb21c",
            "a2d8d2d2359c4a6483437ba335ca59da",
            "a0efa00f518c434ab7bcb04d2a63f280",
            "79fc885bcee6422d9ec103ab43388569",
            "d4dcb489ecd14c96aece33a0f0d50f69",
            "1cc0407eb53f4eb79100447f2c60bf0b",
            "fcb7b564731e4718b1cf3a8006946534",
            "656f58b8d7754317a507ef46fdd0b721",
            "99ccea59cdc14033ba482809384aa288",
            "0b9acf02449d4decae1fa9e86e895e86",
            "2b78357d975a48e0be5e5e1b5c3cd1c2",
            "1edda870424140bdae76ce854924cbc3",
            "a3aec761267b416a81b2bc71a5a0290f",
            "4f22aa3cba794544a16a3a5b8f0b0063",
            "d30071684dbf4caeaaa5a3304a9ad595"
          ]
        },
        "id": "p9DyrkjDEenC",
        "outputId": "8407062f-e372-480e-a81a-8e21b33f7bc9"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ede90b58e6454921807ed5e4add14dfe",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/409 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8bf58fdcd82d40399aaa8b89a65d3533",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "aeadb1c8a93d4c78a13eaa62b12b42ae",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/14.4M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "08a5d57f57bb415f96c3e297b35b669d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "d4dcb489ecd14c96aece33a0f0d50f69",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split: 0 examples [00:00, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['doi', 'chunk-id', 'chunk', 'id', 'title', 'summary', 'source', 'authors', 'categories', 'comment', 'journal_ref', 'primary_category', 'published', 'updated', 'references'],\n",
              "    num_rows: 4838\n",
              "})"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "data = load_dataset(\n",
        "    'jamescalam/llama-2-arxiv-papers-chunked',\n",
        "    split='train'\n",
        ")\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1KGD2k0rFlkn"
      },
      "source": [
        "We will embed and index the documents like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "rXSWtOiRFpw8"
      },
      "outputs": [],
      "source": [
        "data = data.to_pandas()\n",
        "\n",
        "batch_size = 32\n",
        "\n",
        "for i in range(0, len(data), batch_size):\n",
        "    i_end = min(len(data), i+batch_size)\n",
        "    batch = data.iloc[i:i_end]\n",
        "    ids = [f\"{x['doi']}-{x['chunk-id']}\" for i, x in batch.iterrows()]\n",
        "    texts = [x['chunk'] for i, x in batch.iterrows()]\n",
        "    embeds = embed_model.embed_documents(texts)\n",
        "    # get metadata to store in Pinecone\n",
        "    metadata = [\n",
        "        {'text': x['chunk'],\n",
        "         'source': x['source'],\n",
        "         'title': x['title']} for i, x in batch.iterrows()\n",
        "    ]\n",
        "    # add to Pinecone\n",
        "    index.upsert(vectors=zip(ids, embeds, metadata))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z6BhTEMzHvt3",
        "outputId": "08e58e0c-62e7-422c-aa51-530e73b38120"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 384,\n",
              " 'index_fullness': 0.0,\n",
              " 'namespaces': {'': {'vector_count': 4838}},\n",
              " 'total_vector_count': 4838}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHQwEeW9Zps2"
      },
      "source": [
        "## Initializing the Hugging Face Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mElf068NXout"
      },
      "source": [
        "The first thing we need to do is initialize a `text-generation` pipeline with Hugging Face transformers. The Pipeline requires three things that we must initialize first, those are:\n",
        "\n",
        "* A LLM, in this case it will be `meta-llama/Llama-2-13b-chat-hf`.\n",
        "\n",
        "* The respective tokenizer for the model.\n",
        "\n",
        "We'll explain these as we get to them, let's begin with our model.\n",
        "\n",
        "We initialize the model and move it to our CUDA-enabled GPU. Using Colab this can take 5-10 minutes to download and initialize the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 347,
          "referenced_widgets": [
            "767e5e290bd54c1eaff1514e638b3253",
            "e96ac278fa834f968e986ef87fb33847",
            "bdf0f08cbe3b49feb435d001fb35c1a2",
            "202cbc47256f45c2acc8f9fa27d29438",
            "a8e37833d32342ef9eadd59d4a9fe096",
            "998e42ec71854be5a4f625fcdb27cdac",
            "78cf9ce939d94f0d8562e8734dd28c5d",
            "5cabf7fc7b49470c94a35af5e23a3f35",
            "b58780ae351b460f931cc7407e6fd4b4",
            "cb7434f512814641b91c64790b23d27a",
            "4a137d7291464205b22ed6cb70c1569c",
            "42bd48bef0624a359bcd665e962b1b88",
            "68cccb8e845449f2b00cc1dcbe510db0",
            "9acb078fde354d9ba1333796a8b3d442",
            "704ce27da8ff4fb197c21731e0b81fb1",
            "4b62c398507f41e9a11ec5f291f2aa9d",
            "da632eaac44d42928b85918f64b64672",
            "89c5d2c7108a473391bfb00180a03e18",
            "2dc23e6305f349248e257c482f0691eb",
            "d96ed1dad4064ebeb0251c314875f3e3",
            "4af0dad4da024609b5c688f4f4e2bcac",
            "d6611b632ecb4171a192a43af0b6a5b6",
            "ece1c0a5a6a84854ae03612710ef8f67",
            "563a5a8617b6465f9b0bd88f776e8288",
            "b81ebb2c78824ee0bfe0a938720ded58",
            "74d9dc5851764ddf8b03b7844360a3cf",
            "c5007161fb2744f88a04671fcfa46bb2",
            "6c03050710854922a8d633326a941cc1",
            "a38f3199117d4e9880f1e1f1d9de060d",
            "7219fb6d2e18458e96f15e607aa2177c",
            "f8d32c6a61624073b2150d18e913a666",
            "c5be07c6b6e6445ca182bcf331ac8fea",
            "19f5820752f7489ba45bc3a611549f53",
            "bcf34921304b47b4b9fe429877c881e4",
            "7f9c1b4b383a4a4ca849ecefa1f2890f",
            "882631d66c3e4d93b310aca4d02136b8",
            "97bae22c62f14abeba69b67d4927d4bb",
            "a6c59f2acc964c5384c6222705c91757",
            "bf8aee85b4c341d9bd00aced396d7e6c",
            "1239d8a66ddf41a6a72ace3a2ea61e0b",
            "27a654ca834d4992a3e11bd35aa9b080",
            "00055b92f5ac4f229cf01e17229bdb99",
            "314cdfdbe0514294ac50d8e5fcba8e20",
            "e866768f7ba24d15868dfda314957312",
            "abf5ced7f00840ffa2ae74093717d7c0",
            "95ac83fbdbda4f6890d1f3e07102da47",
            "b74c8c3a5fc246ab895010e897691717",
            "ea154f404c054beeb06dfad5d06e76dd",
            "7159e1134b89490fabfa10bc5cda81d8",
            "bee9160bf8ad4a30a6985288003ad6f1",
            "6b519d40110b46c7a5901c80e4620de0",
            "ef62a4856bfb4c98b83cd1af0384e40e",
            "65ceadf08b024b40ae936273a6226d7a",
            "a16cc7a33a274ed58cc4068f8bd617a5",
            "aa951efe549b44afa6c47545c2baa2ef",
            "f9348c2852b64dacb551bd2da1e7b8d4",
            "ae071bca90d2483aad0b47d4a5ea41d3",
            "e971afcc11a14830a38c529f3858b977",
            "f544a1550fa2499e8cc0bfd52a0891f9",
            "fa499f59ae6644e48945e78afb03c051",
            "29e69bf867c54e1ea1f72c701b747425",
            "13fa56fb7d144fffab394ab5b8607198",
            "575f95b142764d039bbcfc882cac69e3",
            "cfde8ae8093c45898115e6827a5c02d1",
            "8a95c8fbcda94854a873cf5926b17dfa",
            "8aa377bcc14047b5a1430c8e01c4346d",
            "34b80cc9892d44c9af7e72f09ddb8f3d",
            "ff9f82bd0f744865a3fa7656c80dc165",
            "57d2726d845d45978ebf7927b6d2ddf8",
            "8fc0a2be5a9a4bc4a546c17195169198",
            "3d3776f13f914c62a9c0cded7bd9934a",
            "773bf6a297254fa3abc9280e094f2510",
            "ad34cd2170604c919489c230a823cfa3",
            "cebc1eb563144e09b663921b01b0db56",
            "93109605eaf34ecd8069b142a9ac9d66",
            "c83691dd175a4eedb2121567ef006d08",
            "4606411db766440097f5521aad10f704",
            "a82e148a249641feaca20e657342dd7e",
            "7dc64c1a8bcf49a8a5d757cd426cbce8",
            "2a179054ed724307b7503c33c3cc2382",
            "b8d1c623c2c54cbaadabce1f6ae07feb",
            "4a726967bcf94f27b390746bb5b0c668",
            "581e85650f2d4926a17496b9f6461b29",
            "803a2c8665914f5583c88c4322384f78",
            "0741219efbd149dba04f45597dfc005c",
            "20a4ffccc9844628971b5380e2bd0f79",
            "a19b9bfaab0c4836aa264cb709fdffe8",
            "5d584484409b499a9a56875438d4960f"
          ]
        },
        "id": "ikzdi_uMI7B-",
        "outputId": "4af21594-4746-4a0a-bb3c-03e6469e56ad"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "767e5e290bd54c1eaff1514e638b3253",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)lve/main/config.json:   0%|          | 0.00/608 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:2193: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "42bd48bef0624a359bcd665e962b1b88",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)fetensors.index.json:   0%|          | 0.00/33.4k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ece1c0a5a6a84854ae03612710ef8f67",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading shards:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "bcf34921304b47b4b9fe429877c881e4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00003.safetensors:   0%|          | 0.00/9.95G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "abf5ced7f00840ffa2ae74093717d7c0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00003.safetensors:   0%|          | 0.00/9.90G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f9348c2852b64dacb551bd2da1e7b8d4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)of-00003.safetensors:   0%|          | 0.00/6.18G [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "34b80cc9892d44c9af7e72f09ddb8f3d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "a82e148a249641feaca20e657342dd7e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)neration_config.json:   0%|          | 0.00/167 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model loaded on cuda:0\n"
          ]
        }
      ],
      "source": [
        "from torch import cuda, bfloat16\n",
        "import transformers\n",
        "\n",
        "model_id = 'meta-llama/Llama-2-13b-chat-hf'\n",
        "\n",
        "device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'\n",
        "\n",
        "# set quantization configuration to load large model with less GPU memory\n",
        "# this requires the `bitsandbytes` library\n",
        "bnb_config = transformers.BitsAndBytesConfig(\n",
        "    load_in_4bit=True,\n",
        "    bnb_4bit_quant_type='nf4',\n",
        "    bnb_4bit_use_double_quant=True,\n",
        "    bnb_4bit_compute_dtype=bfloat16\n",
        ")\n",
        "\n",
        "# begin initializing HF items, need auth token for these\n",
        "hf_auth = 'HF_AUTH_TOKEN'\n",
        "model_config = transformers.AutoConfig.from_pretrained(\n",
        "    model_id,\n",
        "    use_auth_token=hf_auth\n",
        ")\n",
        "\n",
        "model = transformers.AutoModelForCausalLM.from_pretrained(\n",
        "    model_id,\n",
        "    trust_remote_code=True,\n",
        "    config=model_config,\n",
        "    quantization_config=bnb_config,\n",
        "    device_map='auto',\n",
        "    use_auth_token=hf_auth\n",
        ")\n",
        "model.eval()\n",
        "print(f\"Model loaded on {device}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JzX9LqWSX9ot"
      },
      "source": [
        "The pipeline requires a tokenizer which handles the translation of human readable plaintext to LLM readable token IDs. The Llama 2 13B models were trained using the Llama 2 13B tokenizer, which we initialize like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 201,
          "referenced_widgets": [
            "083e65d66f60474bb381b12868163ff7",
            "b45251290a5044d185c9fb6863aa94dd",
            "fb798866bf6f43ad864de64beb3c657e",
            "4f77bbe1fe2844b0b9ef6a2dde40d2f0",
            "6c4407f44ac04a70a5cf1e6de1a46aed",
            "538ceaa7f0ca42348b3b5b069b4e68cd",
            "924cb70e10ca4b678fac24d0b1277e7e",
            "ab35ccb5083144719787d72ba50a8498",
            "9f3ad09509bf4f64b0bf50a432a3fe33",
            "62cbc93876934ab58532a80bb3c344ca",
            "e39bf96541034c2d9dbc5c974c81dfa9",
            "1663eb77bdc84c70b023da44d6853188",
            "097bf5ced65346acb28b2286c0c656bc",
            "b10489a63ddf4c04937bace0c1a44e08",
            "2958b33b47354a4c9bfbee34ba0ddd13",
            "8831015d196e4cbf9611d1a6c6d58359",
            "1b7f6b13f5264f368a872c44748f1daf",
            "854bcd5b8a9143a48851a17bb985c1df",
            "6c214a84bdc64fc2b91c0ef432527b65",
            "2509b718bca64a378666256667e4a0f5",
            "c1b29309972b4254bd95f779412d65ed",
            "b39c8601d78f48baa459035a8ba9e1d3",
            "019b026fbaf4454dbff05cb615f49c4a",
            "5434df7d79f74445b8b07e215c5b311c",
            "3a76e3ff79a247a0a908f86796d0fc7e",
            "fa1fed0564864eff97c8facdee255872",
            "8687782ab6b24285a68ea6edd9931a03",
            "d1a41d5c58ef4fca840814f95efd96e2",
            "ff14feae10684c50a11b722a6025769f",
            "356da37c195f4de0a08e231e42b01af2",
            "a9c565bfc903422bb4aa8aeed6d5bec6",
            "5757e4cfa58649869d05c7b2375054e4",
            "3a258eb8bf0e4fbc94cf591b82855450",
            "e2c8170f85eb45008d833f58141db496",
            "4ba0771a6c9c4ae08d71f6bb04d27c32",
            "f8d09e81e9d145e2a6242d01c387c267",
            "3b06c53464d64f5a90ce1f40724b99ed",
            "df875dd93a3847ff8f404a0e1a6f43d8",
            "9bd4bafae85449f3bcebc85034f48516",
            "19c2043b47d24befb0d7d1c28f9497bb",
            "6f51dc93332b40c7a3c563d1acc8d7a7",
            "a8de21c0072746ec9f36c8f487dd89cd",
            "eeb5fb3890c74c36bdac586ad14d88ac",
            "7833052ecedd49618e85b50c60f2c3c0"
          ]
        },
        "id": "v0iPv1GDGxgT",
        "outputId": "4e93ae82-df68-4c2d-b9a4-eb89d32cde32"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "083e65d66f60474bb381b12868163ff7",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)okenizer_config.json:   0%|          | 0.00/749 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1714: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1663eb77bdc84c70b023da44d6853188",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "019b026fbaf4454dbff05cb615f49c4a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)/main/tokenizer.json:   0%|          | 0.00/1.84M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e2c8170f85eb45008d833f58141db496",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading (\u2026)cial_tokens_map.json:   0%|          | 0.00/414 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
        "    model_id,\n",
        "    use_auth_token=hf_auth\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bNysQFtPoaj7"
      },
      "source": [
        "Now we're ready to initialize the HF pipeline. There are a few additional parameters that we must define here. Comments explaining these have been included in the code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "qAYXi8ayKusU"
      },
      "outputs": [],
      "source": [
        "generate_text = transformers.pipeline(\n",
        "    model=model, tokenizer=tokenizer,\n",
        "    return_full_text=True,  # langchain expects the full text\n",
        "    task='text-generation',\n",
        "    # we pass model parameters here too\n",
        "    temperature=0.0,  # 'randomness' of outputs, 0.0 is the min and 1.0 the max\n",
        "    max_new_tokens=512,  # mex number of tokens to generate in the output\n",
        "    repetition_penalty=1.1  # without this output begins repeating\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8DG1WNTnJF1o"
      },
      "source": [
        "Confirm this is working:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lhFgmMr0JHUF",
        "outputId": "4e6e0b13-24e9-4edd-90ec-0a9e34eae3c0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Explain to me the difference between nuclear fission and fusion.\n",
            "\n",
            "Nuclear fission is a process in which an atomic nucleus splits into two or more smaller nuclei, releasing a large amount of energy in the process. This process typically occurs when an atom is bombarded with a high-energy particle, such as a neutron. When the nucleus splits, it releases a large amount of energy in the form of kinetic energy of the fragments and gamma radiation.\n",
            "\n",
            "Nuclear fusion, on the other hand, is the process by which two or more atomic nuclei combine to form a single, heavier nucleus. This process also releases a large amount of energy, but it does so at much higher temperatures than those required for fission. In order to achieve fusion, the atoms must be heated to incredibly high temperatures, typically over 100 million degrees Celsius.\n",
            "\n",
            "One key difference between fission and fusion is the direction of the energy release. In fission, the energy is released outward from the nucleus, while in fusion, the energy is released inward towards the center of the nucleus. Additionally, fission typically produces more radioactive waste than fusion, as the fission process creates more neutrons that can cause radioactive contamination.\n",
            "\n",
            "Another important difference is the fuel used in each process. Fission typically uses uranium or other heavy elements, while fusion requires lighter elements such as hydrogen or helium. The fuel requirements for fusion are much more demanding, as the reaction requires the fuel to be heated to extremely high temperatures in order to initiate the fusion reaction.\n",
            "\n",
            "In summary, nuclear fission and fusion are both nuclear reactions that release a large amount of energy, but they differ in the direction of energy release, the type of fuel used, and the production of radioactive waste.\n"
          ]
        }
      ],
      "source": [
        "res = generate_text(\"Explain to me the difference between nuclear fission and fusion.\")\n",
        "print(res[0][\"generated_text\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0N3W3cj3Re1K"
      },
      "source": [
        "Now to implement this in LangChain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "-8RxQYwHRg0N"
      },
      "outputs": [],
      "source": [
        "from langchain.llms import HuggingFacePipeline\n",
        "\n",
        "llm = HuggingFacePipeline(pipeline=generate_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "id": "aiW0_FoQWG6J",
        "outputId": "175e879c-fc51-4443-cc4c-9b5d8460b314"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'\\n\\nNuclear fission is a process in which an atomic nucleus splits into two or more smaller nuclei, releasing a large amount of energy in the process. This process typically occurs when an atom is bombarded with a high-energy particle, such as a neutron. When the nucleus splits, it releases a large amount of energy in the form of kinetic energy of the fragments and gamma radiation.\\n\\nNuclear fusion, on the other hand, is the process by which two or more atomic nuclei combine to form a single, heavier nucleus. This process also releases a large amount of energy, but it does so at much higher temperatures than those required for fission. In order to achieve fusion, the atoms must be heated to incredibly high temperatures, typically over 100 million degrees Celsius.\\n\\nOne key difference between fission and fusion is the direction of the energy release. In fission, the energy is released outward from the nucleus, while in fusion, the energy is released inward towards the center of the nucleus. Additionally, fission typically produces more radioactive waste than fusion, as the fission process creates more neutrons that can cause radioactive contamination.\\n\\nAnother important difference is the fuel used in each process. Fission typically uses uranium or other heavy elements, while fusion requires lighter elements such as hydrogen or helium. The fuel requirements for fusion are much more demanding, as the reaction requires the fuel to be heated to extremely high temperatures in order to initiate the fusion reaction.\\n\\nIn summary, nuclear fission and fusion are both nuclear reactions that release a large amount of energy, but they differ in the direction of energy release, the type of fuel used, and the production of radioactive waste.'"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "llm(prompt=\"Explain to me the difference between nuclear fission and fusion.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tv0KxJLvsIa"
      },
      "source": [
        "We still get the same output as we're not really doing anything differently here, but we have now added **Llama 2 13B Chat** to the LangChain library. Using this we can now begin using LangChain's advanced agent tooling, chains, etc, with **Llama 2**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hVu2KHaMLM2M"
      },
      "source": [
        "## Initializing a RetrievalQA Chain"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0l9UNP7LLSXt"
      },
      "source": [
        "For **R**etrieval **A**ugmented **G**eneration (RAG) in LangChain we need to initialize either a `RetrievalQA` or `RetrievalQAWithSourcesChain` object. For both of these we need an `llm` (which we have initialized) and a Pinecone index \u2014 but initialized within a LangChain vector store object.\n",
        "\n",
        "Let's begin by initializing the LangChain vector store, we do it like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "oIbTrJDmpddS"
      },
      "outputs": [],
      "source": [
        "from langchain.vectorstores import Pinecone\n",
        "\n",
        "text_field = 'text'  # field in metadata that contains text content\n",
        "\n",
        "vectorstore = Pinecone(\n",
        "    index, embed_model.embed_query, text_field\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-0dxBmDYpyj3"
      },
      "source": [
        "We can confirm this works like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1WhVonePp0hY",
        "outputId": "9ab6534f-a6cf-492a-fff1-928d4801bc03"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Document(page_content='Ricardo Lopez-Barquilla, Marc Shedro\ufb00, Kelly Michelena, Allie Feinstein, Amit Sangani, Geeta\\nChauhan,ChesterHu,CharltonGholson,AnjaKomlenovic,EissaJamil,BrandonSpence,Azadeh\\nYazdan, Elisa Garcia Anzano, and Natascha Parks.\\n\u2022ChrisMarra,ChayaNayak,JacquelinePan,GeorgeOrlin,EdwardDowling,EstebanArcaute,Philomena Lobo, Eleonora Presani, and Logan Kerr, who provided helpful product and technical organization support.\\n46\\n\u2022Armand Joulin, Edouard Grave, Guillaume Lample, and Timothee Lacroix, members of the original\\nLlama team who helped get this work started.\\n\u2022Drew Hamlin, Chantal Mora, and Aran Mun, who gave us some design input on the \ufb01gures in the\\npaper.\\n\u2022Vijai Mohan for the discussions about RLHF that inspired our Figure 20, and his contribution to the\\ninternal demo.\\n\u2022Earlyreviewersofthispaper,whohelpedusimproveitsquality,includingMikeLewis,JoellePineau,\\nLaurens van der Maaten, Jason Weston, and Omer Levy.', metadata={'source': 'http://arxiv.org/pdf/2307.09288', 'title': 'Llama 2: Open Foundation and Fine-Tuned Chat Models'}),\n",
              " Document(page_content='our responsible release strategy can be found in Section 5.3.\\nTheremainderofthispaperdescribesourpretrainingmethodology(Section2),\ufb01ne-tuningmethodology\\n(Section 3), approach to model safety (Section 4), key observations and insights (Section 5), relevant related\\nwork (Section 6), and conclusions (Section 7).\\n\u2021https://ai.meta.com/resources/models-and-libraries/llama/\\n\u00a7We are delaying the release of the 34B model due to a lack of time to su\ufb03ciently red team.\\n\u00b6https://ai.meta.com/llama\\n\u2016https://github.com/facebookresearch/llama\\n4\\nFigure 4: Training of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc : This process begins with the pretraining ofL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle using publicly\\navailableonlinesources. Followingthis,wecreateaninitialversionof L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle-C/h.sc/a.sc/t.sc throughtheapplication', metadata={'source': 'http://arxiv.org/pdf/2307.09288', 'title': 'Llama 2: Open Foundation and Fine-Tuned Chat Models'}),\n",
              " Document(page_content='Evaluation Results\\nSee evaluations for pretraining (Section 2); \ufb01ne-tuning (Section 3); and safety (Section 4).\\nEthical Considerations and Limitations (Section 5.2)\\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle is a new technology that carries risks with use. Testing conducted to date has been in\\nEnglish, and has notcovered, nor could it coverall scenarios. For these reasons, aswith all LLMs,\\nL/l.sc/a.sc/m.sc/a.sc /two.taboldstyle\u2019s potential outputs cannot be predicted in advance, and the model may in some instances\\nproduceinaccurateorobjectionableresponsestouserprompts. Therefore,beforedeployingany\\napplications of L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle, developers should perform safety testing and tuning tailored to their\\nspeci\ufb01c applications of the model. Please see the Responsible Use Guide available available at\\nhttps://ai.meta.com/llama/responsible-user-guide\\nTable 52: Model card for L/l.sc/a.sc/m.sc/a.sc /two.taboldstyle .\\n77', metadata={'source': 'http://arxiv.org/pdf/2307.09288', 'title': 'Llama 2: Open Foundation and Fine-Tuned Chat Models'})]"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = 'what makes llama 2 special?'\n",
        "\n",
        "vectorstore.similarity_search(\n",
        "    query,  # the search query\n",
        "    k=3  # returns top 3 most relevant chunks of text\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r3zRCEcUqAGC"
      },
      "source": [
        "Looks good! Now we can put our `vectorstore` and `llm` together to create our RAG pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "llyEC13RqF9B"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import RetrievalQA\n",
        "\n",
        "rag_pipeline = RetrievalQA.from_chain_type(\n",
        "    llm=llm, chain_type='stuff',\n",
        "    retriever=vectorstore.as_retriever()\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjY7R4KDKZTw"
      },
      "source": [
        "Let's begin asking questions! First let's try *without* RAG:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "id": "PnBrHM1PT7af",
        "outputId": "69389819-cc4c-4cfd-a7a7-8c4700f230b2"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'\\n\\nAnswer: Llama 2 is a unique and special animal for several reasons. Here are some of the most notable features that make it stand out:\\n\\n1. Size: Llamas are known for their size, and Llama 2 is no exception. It is one of the largest llamas in existence, with some individuals reaching heights of over 6 feet (1.8 meters) at the shoulder and weighing up to 400 pounds (180 kilograms).\\n2. Coat: Llama 2 has a distinctive coat that is soft, fine, and silky to the touch. The coat can be a variety of colors, including white, cream, beige, and brown.\\n3. Temperament: Llama 2 is known for its friendly and docile nature. They are social animals that thrive on human interaction and are often used as therapy animals due to their calm demeanor.\\n4. Intelligence: Llama 2 is highly intelligent and can learn a wide range of tasks, from simple commands like \"sit\" and \"stay\" to more complex tasks like pulling carts or carrying packs.\\n5. Adaptability: Llama 2 is highly adaptable and can survive in a variety of environments, from high-altitude mountains to hot deserts. They are also able to digest a wide range of plants, making them a versatile and valuable animal in many different ecosystems.\\n6. Long lifespan: Llama 2 can live for up to 20 years in captivity, which is longer than many other domesticated animals. This makes them a long-term investment for farmers and breeders.\\n7. Low maintenance: Llama 2 is relatively low maintenance compared to other livestock. They require minimal equipment and infrastructure, and they are easy to handle and care for.\\n8. Dual purpose: Llama 2 is a dual-purpose animal, meaning they can be raised for both their meat and their wool. Their meat is lean and nutritious, while their wool is soft and warm.\\n9. Resistance to disease: Llama 2 is resistant to many diseases that affect other livestock, such as brucellosis and tuberculosis. This makes them a reliable and hard'"
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "llm('what is so special about llama 2?')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v33KdwE_Ua6X"
      },
      "source": [
        "Hmm, that's not what we meant... What if we use our RAG pipeline?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BEndJT3_KYUi",
        "outputId": "b5c24e92-0db2-47a4-bccc-41909142c700"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'query': 'what is so special about llama 2?',\n",
              " 'result': ' Llama 2 is a collection of pretrained and fine-tuned large language models (LLMs) developed and released by GenAI, Meta. The models are optimized for dialogue use cases and outperform open-source chat models on most benchmarks tested. Additionally, they are considered a suitable substitute for closed-source models like ChatGPT, BARD, and Claude.\\n\\nPlease let me know if you need any further information or clarification.'}"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rag_pipeline('what is so special about llama 2?')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1QK9mjspUhC1"
      },
      "source": [
        "This looks *much* better! Let's try some more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 161
        },
        "id": "z65JYMpzUxE3",
        "outputId": "50d25af6-4f85-4879-a23f-3120e651ddad"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "\"\\n\\nI'm looking for information on how the developers of Llama 2 ensured the safety of their users during the development process. Specifically, I'm interested in knowing about any safety measures that were implemented to protect users from potential risks or hazards associated with the use of the platform.\\n\\nHere are some possible answers:\\n\\n1. The developers of Llama 2 conducted thorough risk assessments to identify and mitigate any potential safety risks associated with the platform. This included identifying potential hazards such as data breaches, cyber attacks, and other security risks, and implementing appropriate safeguards to prevent these risks from occurring.\\n2. The platform was designed with user privacy and security in mind, and the developers implemented various measures to protect user data and ensure that it is not compromised. For example, the platform may have implemented encryption techniques to protect user data, or implemented strict access controls to limit who can access user information.\\n3. The developers of Llama 2 implemented a robust testing and quality assurance process to ensure that the platform is stable and reliable before it was released to users. This included conducting extensive testing to identify and fix any bugs or vulnerabilities that could potentially cause harm to users.\\n4. The platform was designed with scalability and reliability in mind, so that it can handle a large number of users and transactions without compromising performance or security. This includes implementing load balancing techniques, redundancy measures, and other infrastructure design best practices to ensure that the platform can handle high volumes of traffic and data without failing.\\n5. The developers of Llama 2 provided clear instructions and guidelines for users on how to safely use the platform, including tips on how to securely store and manage their personal information, and how to avoid common online scams and threats. Additionally, the platform may have provided resources such as customer support and educational materials to help users understand how to use the platform safely and effectively.\""
            ]
          },
          "execution_count": 26,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "llm('what safety measures were used in the development of llama 2?')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EPMfjpmhiJui"
      },
      "source": [
        "Okay, it looks like the LLM with no RAG is less than ideal \u2014 let's stop embarassing the poor LLM and stick with RAG + LLM. Let's ask the same question to our RAG pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9oR8DzztUli2",
        "outputId": "6b7b292a-9e4d-4d77-e34a-e3e82e88b3ee"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'query': 'what safety measures were used in the development of llama 2?',\n",
              " 'result': ' The development of llama 2 included safety measures such as pre-training, fine-tuning, and model safety approaches. Additionally, the authors delayed the release of the 34B model due to a lack of time to sufficiently red team.'}"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rag_pipeline('what safety measures were used in the development of llama 2?')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z2-I1A3MVZJB"
      },
      "source": [
        "A reasonable answer from the RAG pipeline, but it doesn't contain much information \u2014 maybe we can ask more about this, like what is this _\"red team\"_ procedure that delayed the launch of the 34B model?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1GxZJVCBVS8Z",
        "outputId": "090507e0-75be-4ae8-e2aa-a3ee09a27ae1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'query': 'what red teaming procedures were followed for llama 2?',\n",
              " 'result': \" The paper describes the red teaming procedures used for Llama 2. These included creating prompts that might elicit unsafe or undesirable responses from the model, such as those based on sensitive topics or those that could potentially cause harm if the model were to respond inappropriately. The red teaming exercises were performed by a set of experts who evaluated the model's responses and provided feedback on its performance. The paper also mentions that multiple additional rounds of red teaming were performed over several months to measure the robustness of the model as it was released internally.\"}"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rag_pipeline('what red teaming procedures were followed for llama 2?')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vpMTOFzKivfr"
      },
      "source": [
        "Very interesting!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Xg4NQdYiv-P",
        "outputId": "65526e24-2a26-4e8c-dfc5-202a204cda2c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'query': 'how does the performance of llama 2 compare to other local LLMs?',\n",
              " 'result': ' The performance of llama 2 is compared to other local LLMs such as chinchilla and bard in the paper. Specifically, the authors report that llama 2 outperforms these other models on the series of helpfulness and safety benchmarks they tested. Additionally, the authors note that llama 2 appears to be on par with some of the closed-source models, at least on the human evaluations they performed.'}"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rag_pipeline('how does the performance of llama 2 compare to other local LLMs?')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}